Directly connected heat exchanger tube section and coolant-cooled structure

ABSTRACT

A cooling apparatus for an electronics rack is provided which includes an air-to-liquid heat exchanger, one or more coolant-cooled structures and a tube. The heat exchanger, which is associated with the electronics rack and disposed to cool air passing through the rack, includes a plurality of distinct, coolant-carrying tube sections, each tube section having a coolant inlet and a coolant outlet, one of which is coupled in fluid communication with a coolant loop to facilitate flow of coolant through the tube section. The coolant-cooled structure(s) is in thermal contact with an electronic component(s) of the rack, and facilitates transfer of heat from the component(s) to the coolant. The tube connects in fluid communication one coolant-cooled structure and the other of the coolant inlet or outlet of the one tube section, and facilitates flow of coolant directly between that coolant-carrying tube section of the heat exchanger and the coolant-cooled structure.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with Government support under Contract No.DE-EE0002894, awarded by the Department of Energy. Accordingly, the U.S.Government has certain rights in the invention.

BACKGROUND

The power dissipation of integrated circuit chips, and the modulescontaining the chips, continues to increase in order to achieveincreases in processor performance. This trend poses a cooling challengeat both the module and system levels. Increased airflow rates are neededto effectively cool high power modules and to limit the temperature ofthe air that is exhausted into the computer center.

In many large server applications, processors along with theirassociated electronics (e.g., memory, disk drives, power supplies, etc.)are packaged in removable drawer configurations stacked within a rack orframe. In other cases, the electronics may be in fixed locations withinthe rack or frame. Typically, the components are cooled by air moving inparallel airflow paths, usually front-to-back, impelled by one or moreair moving devices (e.g., fans or blowers). In some cases it may bepossible to handle increased power dissipation within a single drawer byproviding greater airflow, through the use of a more powerful air movingdevice or by increasing the rotational speed (i.e., RPMs) of an existingair moving device. However, this approach is becoming problematic at therack level in the context of a computer installation (i.e., datacenter).

The sensible heat load carried by the air exiting the rack is stressingthe ability of the room air-conditioning to effectively handle the load.This is especially true for large installations with “server farms” orlarge banks of computer racks close together. In such installations,liquid cooling (e.g., water cooling) is an attractive technology tomanage the higher heat fluxes. The liquid absorbs the heat dissipated bythe components/modules in an efficient manner. Typically, the heat isultimately transferred from the liquid to an outside environment,whether air or other liquid.

BRIEF SUMMARY

In one aspect, the shortcomings of the prior art are overcome andadditional advantages are provided through the provision of a coolingapparatus for an electronics rack. The cooling apparatus includes anair-to-liquid heat exchanger, at least one coolant-cooled structure, anda tube connecting in fluid communication one coolant-cooled structure ofthe at least one coolant-cooled structure and one coolant-carrying tubesection of the air-to-liquid heat exchanger. The air-to-liquid heatexchanger is associated with the electronics rack and disposed to coolat least a portion of air passing through the electronics rack, whereinair moves through the electronics rack from an air inlet side to an airoutlet side thereof, and the air-to-liquid heat exchanger comprises aplurality of distinct, coolant-carrying tube sections. Eachcoolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections has a coolant inlet and a coolant outlet.One of the coolant inlet or the coolant outlet is coupled in fluidcommunication with a coolant loop to facilitate flow of coolant throughthe coolant-carrying tube section. The at least one coolant-cooledstructure is in thermal contact with at least one electronic componentof the electronics rack, and the at least one coolant-cooled structurefacilitates transfer of heat from the at least one electronic componentto the coolant. The tube connects in fluid communication the onecoolant-cooled structure of the at least one coolant-cooled structureand the other of the coolant inlet or the coolant outlet of the onecoolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections of the air-to-liquid heat exchanger. Thetube facilitates flow of coolant directly between the onecoolant-carrying tube section of the air-to-liquid heat exchanger andthe one coolant-cooled structure of the at least one coolant-cooledstructure.

In another aspect, a cooled electronic system is provided which includesan electronics rack and a cooling apparatus for facilitating cooling ofthe electronics rack. The electronics rack includes an air inlet sideand an air outlet side for respectively enabling ingress and egress ofair through the electronics rack, and at least one electronic componentto be cooled. The cooling apparatus includes an air-to-liquid heatexchanger, at least one coolant-cooled structure, and a tube connectingin fluid communication one coolant-cooled structure of the at least onecoolant-cooled structure and one coolant-carrying tube section of theair-to-liquid heat exchanger. The air-to-liquid heat exchanger isassociated with the electronics rack and disposed to cool at least aportion of the air passing through the electronics rack. Eachcoolant-carrying tube section of a plurality of distinct,coolant-carrying tube sections of the air-to-liquid heat exchanger has acoolant inlet and a coolant outlet. One of the coolant inlet or thecoolant outlet is coupled in fluid communication with a coolant loop tofacilitate flow of coolant through the coolant-carrying tube section.The at least one coolant-cooled structure is in thermal contact with theat least one electronic component of the electronics rack, andfacilitates transfer of heat from the at least one electronic componentto the coolant. The tube connects in fluid communication the onecoolant-cooled structure of the at least one coolant-cooled structureand the other of the coolant inlet or the coolant outlet of the onecoolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections of the air-to-liquid heat exchanger. Thetube facilitates flow of coolant directly between the onecoolant-carrying tube section of the air-to-liquid heat exchanger andthe one coolant-cooled structure of the at least one coolant-cooledstructure.

In a further aspect, a method of fabricating a cooling apparatus forfacilitating cooling of an electronics rack is provided. The methodincludes: associating an air-to-liquid heat exchanger with theelectronics rack and disposing the air-to-liquid heat exchanger to coolat least a portion of air passing through the electronics rack, whereinair moves through the electronics rack from an air inlet side to an airoutlet side thereof, and the air-to-liquid heat exchanger comprises aplurality of distinct, coolant-carrying tube sections, eachcoolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections comprising a coolant inlet and a coolantoutlet, one of the coolant inlet or the coolant outlet being coupled influid communication with a coolant loop to facilitate flow of coolantthrough the coolant-carrying tube section; providing at least onecoolant-cooled structure in thermal contact with at least one electroniccomponent of the electronics rack, the at least one coolant-cooledstructure facilitating transfer of heat from the at least one electroniccomponent to the coolant; and providing a tube connecting in fluidcommunication one coolant-cooled structure of the at least onecoolant-cooled structure and the other of the coolant inlet or thecoolant outlet of the one coolant-carrying tube section of the pluralityof distinct, coolant-carrying tube sections of the air-to-liquid heatexchanger, the tube facilitating flow of coolant directly between theone coolant-carrying tube section of the air-to-liquid heat exchangerand the one coolant-cooled structure of the at least one coolant-cooledstructure.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

One or more aspects of the present invention are particularly pointedout and distinctly claimed as examples in the claims at the conclusionof the specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 depicts one embodiment of a conventional raised floor layout ofan air-cooled data center;

FIG. 2A is a cross-sectional plan view of one embodiment of anelectronics rack with an attached air-to-liquid heat exchanger coolingair passing through the electronics rack, in accordance with one or moreaspects of the present invention;

FIG. 2B is a cross-sectional plan view of another embodiment of anelectronics rack with an attached air-to-liquid heat exchanger coolingair passing through the electronics rack, in accordance with one or moreaspects of the present invention;

FIG. 3 depicts one embodiment of a data center with a coolantdistribution unit facilitating liquid-cooling of one or moreliquid-cooled electronics racks of the data center, in accordance withone or more aspects of the present invention;

FIG. 4 depicts an alternate embodiment of a cooling apparatus andliquid-cooled electronics rack, in accordance with one or more aspectsof the present invention;

FIG. 5A is an elevational view of one embodiment of a cooled electronicsystem comprising an electronics rack and an airflow director configuredto redirect airflow exhausting from the electronics rack at the airoutlet side through an airflow return pathway back towards the air inletside of the electronics rack, in accordance with one or more aspects ofthe present invention;

FIG. 5B is a cross-sectional plan view of one embodiment of the cooledelectronic system of FIG. 5A, depicting one embodiment of a coolingapparatus for the electronics rack, and illustrating airflow within theclosed loop airflow pathway through the electronics rack and airflowdirector, in accordance with one or more aspects of the presentinvention;

FIG. 6 is a plan view of another embodiment of an electronic systemlayout of an electronic subsystem with a coolant-cooled structureassociated therewith comprising, for example, multiple liquid-cooledcold plates and multiple liquid-cooled cold rails coupled in fluidcommunication, in accordance with one or more aspects of the presentinvention;

FIG. 7A is a more detailed, elevational view of one embodiment of theliquid-cooled electronics rack of FIGS. 4 & 5A, and illustratingrack-level coolant distribution structures, in accordance with one ormore aspects of the present invention;

FIG. 7B is a partial depiction of a more detailed embodiment of therack-level coolant distribution structures illustrated in FIG. 7A, inaccordance with one or more aspects of the present invention;

FIG. 8 depicts another embodiment of a cooled electronic systemcomprising a cooling apparatus and a liquid-cooled electronics rack, inaccordance with one or more aspects of the present invention;

FIG. 9A is a partial, detailed elevational view of one embodiment offluid connections between the air-to-liquid heat exchanger of thecooling apparatus and the liquid-cooled structures of the liquid-cooledelectronics rack, in accordance with one or more aspects of the presentinvention;

FIG. 9B is a more detailed, partial elevational view of the structuresof FIG. 9A, in accordance with one or more aspects of the presentinvention; and

FIG. 10 is an elevational view of one embodiment of the cooledelectronic system of FIGS. 8-9B, with one or more covers removed toillustrate the air-to-liquid heat exchanger and the liquid-cooledelectronics rack of the cooled electronic system, in accordance with oneor more aspects of the present invention.

DETAILED DESCRIPTION

As used herein, the terms “electronics rack”, “rack-mounted electronicequipment”, and “rack unit” are used interchangeably, and unlessotherwise specified include any housing, frame, rack, compartment, bladeserver system, etc., having one or more heat generating components of acomputer system or electronic system, and may be, for example, astand-alone computer processor having high, mid or low end processingcapability. In one embodiment, an electronics rack may comprise aportion of an electronic system, a single electronic system or multipleelectronic systems, for example, in one or more sub-housings, blades,books, drawers, nodes, compartments, etc., having one or moreheat-generating electronic components disposed therein. An electronicsystem(s) within an electronics rack may be movable or fixed relative tothe electronics rack, with rack-mounted electronic drawers and blades ofa blade center system being two examples of electronic systems (orsubsystems) of an electronics rack to be cooled.

“Electronic component” refers to any heat-generating electroniccomponent of, for example, a computer system or other electronic systemrequiring cooling. By way of example, an electronic component maycomprise one or more integrated circuit dies, and/or other electronicdevices to be cooled, such as one or more electronics cards comprising aplurality of memory modules (such as one or more dual in-line memorymodules (DIMMs)).

Further, as used herein, the terms “liquid-cooled cold plate” and“liquid-cooled cold rail” refer to thermally conductive structureshaving one or more channels (or passageways) formed therein or passingtherethrough, which facilitate the flow of liquid coolant through thestructure. A “liquid-cooled structure” may comprise, for example, one ormore liquid-cooled cold plates and/or one or more liquid-cooled coldrails coupled in fluid communication and positioned in thermal contactwith one or more electronic components to be cooled (for example, of anassociated electronic subsystem). In one example, tubing is providedextending through or coupling in fluid communication the liquid-cooledcold plates and/or the liquid-cooled cold rails of the liquid-cooledstructure.

An “air-to-liquid heat exchanger” or “air-to-liquid heat exchangeassembly” means any heat exchange mechanism characterized as describedherein through which liquid coolant can circulate; and includes, one ormore discrete air-to-liquid heat exchangers coupled either in series orin parallel. An air-to-liquid heat exchanger may comprise, for example,one or more coolant flow paths, formed of thermally conductive tubing(such as copper or other tubing) in thermal or mechanical contact with aplurality of air-cooled cooling fins. Size, configuration andconstruction of the air-to-liquid heat exchanger can vary withoutdeparting from the scope of the invention disclosed. Still further,“data center” refers to a computer installation containing one or moreelectronics racks to be cooled. As a specific example, a data center maycomprise one or more rows of rack-mounted computer units, such as serverunits.

One example of coolant used within the cooled electronic systemsdisclosed herein is water. However, the concepts presented are readilyadapted to use with other types of coolant. For example, the coolant maycomprise a brine, a fluorocarbon liquid, a liquid metal, or othersimilar coolant, or refrigerant, while still maintaining the advantagesand unique features of the present invention.

Reference is made below to the drawings, which are not drawn to scalefor reasons of understanding, wherein the same reference numbers usedthroughout different figures designate the same or similar components.

FIG. 1 depicts a raised floor layout of an air cooled data center 100typical in the prior art, wherein multiple electronics racks 110 aredisposed in one or more rows. A data center such as depicted in FIG. 1may house several hundred, or even several thousand microprocessors. Inthe arrangement illustrated, chilled air enters the computer room viaperforated floor tiles 160 from a supply air plenum 145 defined betweenthe raised floor 140 and a base or sub-floor 165 of the room. Cooled airis taken in through louvered covers at air inlet sides 120 of theelectronics racks and expelled through the back (i.e., air outlet sides130) of the electronics racks. Each electronics rack 110 may have one ormore air moving devices (e.g., fans or blowers) to provide forcedinlet-to-outlet airflow to cool the electronic devices within the rackunit. The supply air plenum 145 provides conditioned and cooled air tothe air-inlet sides of the electronics racks via perforated floor tiles160 disposed in a “cold” aisle of the computer installation. Theconditioned and cooled air is supplied to plenum 145 by one or more airconditioning units 150, also disposed within the data center 100. Roomair is taken into each air conditioning unit 150 near an upper portionthereof. This room air may comprise, in part, exhausted air from the“hot” aisles of the computer installation defined, for example, byopposing air outlet sides 130 of the electronics racks 110.

Due to ever-increasing air flow requirements through electronics racks,and the limits of air distribution within a typical data centerinstallation, liquid-based cooling is being combined with conventionalair-cooling. FIGS. 2A-4 illustrate various embodiments of a data centerimplementation employing a liquid-based cooling system.

FIG. 2A depicts one rack-level liquid-cooling solution which utilizeschilled facility water to remove heat from the computer installationroom, thereby transferring the cooling burden from the air-conditioningunit(s) to the building's chilled water coolers. The embodiment depictedin FIG. 2A is described in detail in commonly assigned, U.S. Pat. No.6,775,137. Briefly summarized, facility-chilled water 200 circulatesthrough one or more liquid-to-liquid heat exchangers 210, coupled via asystem coolant loop 211, to individual electronics racks 220 within thecomputer room. Rack unit 220 includes one or more air-moving devices 230for moving air flow from an air inlet side to an air outlet side acrossone or more drawer units 240 containing heat-generating electroniccomponents to be cooled. In this embodiment, a front cover 250 attachedto the rack covers the air inlet side, a back cover 255 attached to therack covers the air outlet side, and a side car disposed adjacent to(and/or attached to) the rack includes a heat exchanger 260 for coolingair circulating through the rack unit. Further, in this embodiment, theliquid-to-liquid heat exchangers 210 are multiple computer roomwater-conditioning (CRWC) units which are coupled to receive buildingchilled facility water 200. The building chilled facility water is usedto cool the system coolant within system coolant loop 211, which iscirculating through air-to-liquid heat exchanger 260. The rack unit inthis example is assumed to comprise a substantially enclosed housing,wherein the same air circulates through the housing that passes acrossthe air-to-liquid heat exchanger 260. In this manner, heat generatedwithin the electronics rack is removed from the enclosed housing via thesystem coolant loop, and transferred to the facility coolant loop forremoval from the computer installation room.

FIG. 2B illustrates another embodiment of a rack-level, liquid-coolingsolution which utilizes chilled facility water to remove heat from thecomputer room installation, thereby transferring at least a portion ofthe cooling burden from the air-conditioning unit(s) to the building'schilled water coolers. This embodiment is described, for example, inU.S. Pat. No. 7,385,810 B2. Briefly summarized, due to theever-increasing airflow requirements through electronics racks, andlimits of air distribution within the typical computer roominstallation, recirculation problems within the room may occur. Forexample, hot air recirculation may occur from the air outlet side of anelectronics rack back to the cold air aisle defined by, for example,opposing air inlet sides of electronics racks within the data center.This recirculation can occur because the conditioned air suppliedthrough the tiles is typically only a fraction of the airflow rateforced through the electronics rack by the air-moving devices disposedtherein. This can be due, for example, to limitations on the tile sizes(or diffuser flow rates). The remaining fraction of the supply of inletside air is often made up by ambient room air through recirculation.This recirculation flow is often very complex in nature, and can lead tosignificantly higher rack unit inlet temperatures than might beexpected.

The recirculation of hot exhaust air from the hot aisle of the computerroom installation to the cold aisle can be detrimental to theperformance and reliability of the computer system(s) or electronicsystem(s) within the racks. Data center equipment is typically designedto operate with rack air inlet temperatures in the 18-35° C. range. Fora raised floor layout such as depicted in FIG. 1, however, temperaturescan range from 15-20° C. at the lower portion of the rack, close to thecooled air input floor vents, to as much as 45-50° C. at the upperportion of the electronics rack, where the hot air can form aself-sustaining recirculation loop. Since the allowable rack heat loadis limited by the rack inlet air temperature at the “hot” part, thistemperature distribution correlates to a lower processing capacity.Also, computer installation equipment almost always represents a highcapital investment to the customer. Thus, it is of significantimportance, from a product reliability and performance view point, andfrom a customer satisfaction and business perspective, to maintain thetemperature of the inlet air substantially uniform. The efficientcooling of such computer and electronic systems, and the amelioration oflocalized hot air inlet temperatures to one or more rack units due torecirculation of air currents, are addressed by the apparatuses andmethods disclosed herein.

As noted, FIG. 2B depicts another embodiment of a cooled electronicsystem, generally denoted 270, in accordance with an aspect of thepresent invention. In this embodiment, electronic system 270 includes anelectronics rack 272 having an inlet door 280 and an outlet door 282,which have openings to allow for the ingress and egress of external airfrom the inlet side to the outlet side of the electronics rack 272. Thesystem further includes at least one air-moving device 274 for movingexternal air across at least one electronic subsystem 276 positionedwithin the electronics rack. Disposed within, for example, outlet door282 is a heat exchange assembly 284. Heat exchange assembly 284 includesan air-to-liquid heat exchanger through which the inlet-to-outletairflow through the electronics rack passes. A computer roomwater-conditioner (CRWC) 286 is used to buffer heat exchange assembly284 from the building utility (or local chiller) coolant 288, which isprovided as input to CRWC 286. The CRWC 286 provides, for example,system water (or system coolant) to heat exchange assembly 284. Heatexchange assembly 284 removes heat from the exhausting inlet-to-outletairflow through the electronics rack for transfer via the system water(or coolant) to CRWC 286. Advantageously, providing a heat exchangeassembly with an air-to-liquid heat exchanger such as disclosed hereinat the outlet door cover (or at the inlet door cover) of one or moreelectronics racks in a computer installation can, in normal operation,significantly reduce heat loads on existing air-conditioning unitswithin the computer installation, and facilitate the cooling of therack-mounted electronics units.

FIG. 3 depicts another embodiment of a rack-level, liquid-coolingsolution, which again uses chilled facility water to remove heat fromthe computer installation room, thereby transferring the cooling burdenfrom the air-conditioning unit(s) to the building's chilled watercoolers. In this implementation, one embodiment of a coolantdistribution unit 300 for a data center is illustrated. Within coolantdistribution unit 300 is a power/control element 312, areservoir/expansion tank 313, a liquid-to-liquid heat exchanger 314, apump 315 (often accompanied by a redundant second pump), facility waterinlet 316 and outlet 317 supply pipes, a supply manifold 318 supplyingwater or system coolant to the electronics racks 110 via couplings 320and lines 322, and a return manifold 319 receiving water or systemcoolant from the electronics racks 110, via lines 323 and couplings 321.Each electronics rack includes (in one example) a power/control unit 330for the electronics rack, multiple electronic systems or subsystems 340,a system coolant supply manifold 350, and a system coolant returnmanifold 360. As shown, each electronics rack 110 is disposed on raisedfloor 140 of the data center with lines 322 providing system coolant tosystem coolant supply manifolds 350 and lines 323 facilitating return ofsystem coolant from system coolant return manifolds 360 being disposedin the supply air plenum beneath the raised floor.

In the embodiment illustrated, system coolant supply manifold 350provides system coolant to cooling apparatuses disposed within theelectronic systems or subsystems (for example, to liquid-cooled coldplates or cold rails) via flexible hose connections 351, which aredisposed between the supply manifold and the respective electronicsystems within the rack. Similarly, system coolant return manifold 360is coupled to the electronic systems via flexible hose connections 361.Quick connect couplings may be employed at the interface betweenflexible hoses 351, 361 and the individual electronic systems. By way ofexample, these quick connect couplings may comprise various types ofcommercially available couplings, such as those available from ColderProducts Company, of St. Paul, Minn., USA, or Parker Hannifin, ofCleveland, Ohio, USA.

Although not shown, electronics rack 110 may also include anair-to-liquid heat exchanger, for example, disposed at an air outletside thereof, which also receives system coolant from the system coolantsupply manifold 350 and returns system coolant to the system coolantreturn manifold 360.

FIG. 4 illustrates another embodiment of a liquid-cooled electronicsrack and cooling system therefor, in accordance with one or more aspectsof the present invention. In this embodiment, the electronics rack 400has a side car structure 410 associated therewith or attached thereto,which includes an air-to-liquid heat exchanger 415 through which aircirculates from an air outlet side of electronics rack 400 towards anair inlet side of electronics rack 400, for example, in a closed looppath in a manner similar to that illustrated above in connection withthe cooling implementation of FIG. 2. In this example, the coolingsystem comprises an economizer-based, warm-liquid coolant loop 420,which comprises multiple coolant tubes (or lines) connecting, in theexample depicted, air-to-liquid heat exchanger 415 in series fluidcommunication with a coolant supply manifold 430 associated withelectronics rack 400, and connecting in series fluid communication, acoolant return manifold 431 associated with electronics rack 400, acooling unit 440 of the cooling system, and air-to-liquid heat exchanger415.

As illustrated, coolant flowing through warm-liquid coolant loop 420,after circulating through air-to-liquid heat exchanger 415, flows viacoolant supply plenum 430 to one or more electronic systems ofelectronics rack 400, and in particular, one or more cold plates and/orcold rails 435 associated with the electronic systems, before returningvia coolant return manifold 431 to warm-liquid coolant loop 420, andsubsequently to a cooling unit 440 disposed (for example) outdoors fromthe data center. In the embodiment illustrated, cooling unit 440includes a filter 441 for filtering the circulating liquid coolant, anair-to-liquid heat exchanger 442 for removing heat from the liquidcoolant, and a pump 443 for returning the liquid coolant throughwarm-liquid coolant loop 420 to air-to-liquid heat exchanger 415, andsubsequently to the liquid-cooled electronics rack 400. By way ofexample, hose barb fittings 450 and quick disconnect couplings 455 maybe employed to facilitate assembly or disassembly of warm-liquid coolantloop 420.

In one example of the warm coolant-cooling approach of FIG. 4, ambienttemperature might be 30° C., and coolant temperature 35° C. leaving theair-to-liquid heat exchanger 442 of the cooling unit. The cooledelectronic system depicted thus facilitates a chiller-less data center.Advantageously, such a liquid-cooling solution provides highly energyefficient cooling of the electronic systems of the electronics rack,using liquid (e.g., water), that is cooled via circulation through theair-to-liquid heat exchanger located outdoors (i.e., a dry cooler) withexternal ambient air being pumped through the dry cooler. Note that thiswarm coolant-cooling approach of FIG. 4 is presented by way of exampleonly. In alternate approaches, cold coolant-cooling could be substitutedfor the cooling unit 440 depicted in FIG. 4. Such cold coolant-coolingmight employ building chilled facility coolant to cool the liquidcoolant flowing through the liquid-cooled electronics rack, andassociated air-to-liquid heat exchanger (if present), in a manner suchas described above in connection with FIGS. 2A & 3.

FIGS. 5A-7B depict in greater detail one embodiment of a cooledelectronic system comprising a liquid-cooled electronics rack, such asdepicted in FIG. 4, in accordance with one or more aspects of thepresent invention. In FIG. 5A, an electronics rack 500 is depictedsurrounded by an airflow director 501 configured for the electronicsrack. Airflow director 501 is configured to redirect airflow exhaustingfrom the electronics rack at the air outlet side thereof via an airflowreturn pathway passing through, for example, a sidecar 502 of theairflow director, back towards the air inlet side of the electronicsrack. The air-to-liquid heat exchange assembly (not shown) is disposedwithin the airflow return pathway defined between electronics rack 500and airflow director 501 for cooling redirected airflow exhausting fromthe air outlet side of the electronics rack before returning to the airinlet side of the electronics rack. Note that in this embodiment, theairflow director and electronics rack together define the closed-loopairflow return pathway, and substantially all heat generated within theelectronics rack, that is rejected to the recirculating air flow, isremoved via the coolant passing through the air-to-liquid heat exchangerdisposed within the airflow return pathway and the one or morecoolant-cooled structures associated with the electronic subsystems.

FIG. 5B is a cross-sectional plan view of one embodiment of the cooledelectronic system depicted in FIGS. 4 & 5A. By way of example, thecooled electronic system of FIG. 5B is an economizer-based,warm-liquid-cooling system for highly efficient cooling of theelectronic systems within the electronics rack using a liquid that iscooled via circulation through an outdoor air-to-liquid heat exchanger(see FIG. 4), with external ambient air being pumped through the outdoorair-to-liquid heat exchanger. FIG. 5B illustrates one embodiment of anelectronic system (or subsystem) 510 component layout, wherein one ormore air-moving devices 512 provide forced airflow 513 to cool multiplecomponents 514 within electronic system 510. Cool air is taken inthrough a front 515 and exhausted out a back 516 of the electronicsystem (or drawer). The multiple components to be cooled include, forexample, multiple processor modules to which a liquid-cooled structureis coupled in thermal contact. In the example depicted, theliquid-cooled structure comprises multiple liquid-cooled cold plates 520coupled in series fluid communication between a rack-level coolantsupply manifold 521 and a rack-level coolant return manifold 522.

Also depicted in FIG. 5B is an air-to-liquid heat exchanger 530 disposedwithin airflow return pathway 531 defined by airflow director 501 andelectronics rack 500. In this embodiment, air-to-liquid heat exchanger530 is disposed within the airflow return pathway 531 along a side ofelectronics rack 500 extending transverse to air inlet side 515 and airoutlet side 516 thereof. The air-to-liquid heat exchanger 530 isconfigured and disposed within the airflow return pathway for coolingredirected airflow exhausting from the air outlet side 516 ofelectronics rack 500 before returning to the air inlet side 515 ofelectronics rack 500. A coolant supply manifold 532 and a coolant returnmanifold 533 are associated with air-to-liquid heat exchanger 530 forfacilitating the flow of coolant through the heat exchanger. Asillustrated, a coolant flow connection 540 couples in fluidcommunication coolant return manifold 533 and rack-level coolant supplymanifold 521. Thus, in this embodiment, coolant from the coolant loopflows through coolant supply manifold 532, air-to-liquid heat exchanger530, coolant return manifold 533, rack-level coolant supply manifold521, one or more liquid-cooled structures 519, and thereafter,rack-level coolant return manifold 522, before returning to the outdoorair-to-liquid heat exchanger (dry cooler, not shown) for rejection ofheat to, for example, external ambient air.

Those skilled in the art will note that the embodiment depicted in FIG.5B comprises both liquid-cooled and air-cooled devices. The air heatload is cooled using an air-to-liquid heat exchanger disposed within theairflow return pathway defined by the airflow director coupled to theelectronics rack. Rack-level coolant distribution includes a rack-levelcoolant supply manifold and a rack-level coolant return manifold, eachof which comprises, a vertical plenum to which node-level flexible hosessupply and extract liquid from the liquid-cooled structures associatedwith the individual nodes (see, for example, FIGS. 7A-7B).

FIG. 6 illustrates another embodiment of a cooled electronic system 510′component layout, wherein one or more air-moving devices 600 provideforced air flow 601 to cool multiple components 610 within electronicsystem 510′. Cool air is taken in through a front 602 and exhausted outa back 603 of the electronic system (or drawer). The multiple componentsto be cooled include, for example, multiple processor modules to whichliquid-cooled cold plates 620 (of a liquid-cooled structure) arecoupled, as well as multiple arrays 631, 632 of electronics cards 630(e.g., memory modules such as dual in-line memory modules (DIMMs)),which are to be thermally coupled to one or more liquid-cooled coldrails 625 of the liquid-cooled structure associated with the electronicsystem. As used herein “thermally coupled” refers to a physical thermaltransport path being established between components, for example,between an electronics card and a liquid-cooled cold rail for theconduction of heat from one to the other.

The illustrated liquid-based cooling approach further includes multiplecoolant-carrying tubes connecting in fluid communication liquid-cooledcold plates 620 and liquid-cooled cold rails 625. These coolant-carryingtubes comprise (for example), a coolant supply tube 640, multiple bridgetubes 641, and a coolant return tube 642. In the embodiment illustrated,bridge tubes 641 connect one liquid-cooled cold rail 625 in seriesbetween the two liquid-cooled cold plates 620, and connect in paralleltwo additional liquid-cooled cold rails 625 between the secondliquid-cooled cold plate 620 and the coolant return tube 642. Note thatthis configuration is provided by way of example only. The conceptsdisclosed herein may be readily adapted to use with variousconfigurations of liquid-cooled structure and cooled electronic systemlayouts. Note also, that as depicted herein, the liquid-cooled coldrails are elongate, thermally conductive structures comprising one ormore channels through which liquid coolant passes, for example, via oneor more tubes extending through the structures. The liquid-cooled coldrails are disposed, in the embodiment illustrated, at the ends of thetwo arrays (or banks) 631, 632 of electronics cards 630, and multiplethermal spreaders are provided coupling in thermal communicationelectronics cards 630 and liquid-cooled cold rails 625.

As noted, FIGS. 7A & 7B depict in greater detail one embodiment of aliquid-cooled electronics rack, such as depicted in FIGS. 4-6, inaccordance with one or more aspects of the present invention. In thisimplementation, the liquid-cooled electronics rack 700 comprises aplurality of electronic systems 710, within which one or more electroniccomponents are to be liquid-cooled via, for example, one or more coldplates or cold rails of a liquid-cooled structure, as described below.The cooling system includes coolant loop 725 coupled in fluidcommunication with rack-level coolant supply manifold 720 and rack-levelcoolant return manifold 730, both of which may comprisevertically-oriented manifolds attached to liquid-cooled electronics rack700. In this embodiment, the rack-level coolant distribution systemfurther includes individual node-level supply hoses 740 supplyingcoolant from coolant supply manifold 720 to the cold plates and coldrails within the electronic systems 710. As shown in FIG. 7B, coolantsupply manifold 740 may be (in one embodiment) a vertically-orientedmanifold with a plurality of coupling connections 711 disposed along themanifold, one for each electronic system 710 having one or moreelectronic components to be liquid-cooled. Coolant leaves the individualelectronic systems 710 via node-level return hoses 750, which couple theindividual liquid-cooled structures associated with the electronicsystems (or nodes) to coolant return manifold 730, and hence, to coolantloop 725. In the embodiment illustrated in FIG. 4, relativelywarm-liquid coolant, such as water, is supplied from the cooling unit,either directly, or through one or more air-to-liquid heat exchanger(s)415 (of FIG. 4), and hot coolant is returned via the coolant returnmanifold to the cooling unit. In one embodiment of the rack-levelcoolant distribution system illustrated in FIGS. 7A & 7B, the node-levelsupply and return hoses 740, 750 are flexible hoses.

Another embodiment of a cooled electronic system comprising aliquid-cooled electronics rack and cooling apparatus therefor, inaccordance with one or more aspects of the present invention, isdepicted in FIGS. 8-10. Referring collectively to these figures, in oneembodiment, the electronics rack 800 has a sidecar structure 810associated therewith (or attached thereto), which includes anair-to-liquid heat exchanger 815 through which air circulates from anair outlet side of electronics rack 800 towards an air inlet side ofelectronics rack 800, for example, in a closed loop path in a mannersimilar to that described above in connection with the coolingimplementations of FIGS. 2A, 4, 5A & 5B. Note that in an alternateembodiment, the air-to-liquid heat exchanger 815 could be disposed atone or more of the air inlet side or air outlet side of the electronicsrack, for example, as described above in connection with FIG. 2B.

In the example of FIGS. 8-10, the cooling apparatus (or system)comprises an economizer-based, warm-liquid coolant loop 820 connectingin fluid communication a coolant unit 840 to the cooled electronicsystem. In the embodiment depicted in FIG. 8, the cooled electronicsystem comprises a single coolant supply manifold 830 associated withair-to-liquid heat exchanger 815 and a single coolant return manifold831 comprising a rack-level coolant return manifold associated withelectronics rack 800. Also, in accordance with this embodiment of thecooled electronic system, the air-to-liquid heat exchanger 815 isdivided into a plurality of distinct (e.g., separate), coolant-carryingtube sections 817, each of which may comprise a coolant inlet and acoolant outlet.

By way of example, in one embodiment, the coolant inlet of eachdistinct, coolant-carrying tube section 817 is coupled in fluidcommunication with coolant supply manifold 830 to receive coolanttherethrough from coolant loop 820. The coolant outlets of eachcoolant-carrying tube section 817 are coupled via a respective tubing819 in fluid communication with a liquid-cooled structure 835 disposedwithin or associated with a respective electronic system (or subsystem)836. By way of example, each liquid-cooled structure 835 may compriseone or more liquid-cooled cold plates or cold rails, and have anydesired component layout. By way of example, reference the layoutsdescribed above in connection with FIGS. 5B & 6. Coolant exhausted fromthe individual electronic systems (or nodes) 836 is returned viarack-level coolant return manifold 831 to coolant loop 820.Advantageously, in this implementation, only two coolant manifolds areemployed, one associated with the air-to-liquid heat exchanger 815, andthe other associated with the electronics rack 800, which advantageouslysaves costs in implementation. The plurality of tubes 819 may comprise aplurality of flexible hoses, and quick connects 855 or hose barbfittings 850 may be employed throughout the cooled electronic system tofacilitate coupling of the various coolant lines.

Note that in the example depicted, there is a one-to-one correspondencebetween the number of coolant-carrying tube sections 817 withinair-to-liquid heat exchanger 815 and the number of electronic systems(or subsystems) 836 or associated liquid-cooled structure 835 withinelectronics rack 800. This is by way of example only. In an alternateimplementation, there may be (for example) less coolant-carrying tubesections 817 within the air-to-liquid heat exchanger than the number ofelectronic systems (or nodes) 836 or associated coolant-cooledstructures 835 within the electronics rack. In such an implementation,only selected electronic systems 836 might include coolant-cooledstructures 835 coupled to receive coolant from the air-to-liquid heatexchanger, with the remaining systems being air-cooled, as explainedherein.

As illustrated, coolant flowing through warm-liquid coolant loop 820,after passing through the respective distinct, coolant-carrying tubesections 817, flows via tubing 819 to one or more electronic systems 836of electronics rack 800, and in particular, to one or morecoolant-cooled structures 835 associated with the electronic systems836, before returning via rack-level coolant return manifold 831 towarm-liquid coolant loop 820, and subsequently, to cooling unit 840disposed (for example) outdoors from the data center. In the embodimentillustrated, cooling unit 840 includes a filter 841 for filtering thecirculating liquid coolant, an air-to-liquid heat exchanger 842 forremoving heat from the liquid coolant, and a pump 843 for returning theliquid coolant through warm-liquid coolant loop 820 to coolant supplymanifold 830 for again passing through the plurality of distinct,coolant-carrying tube sections 817 of air-to-liquid heat exchanger 815,and subsequently, to the coolant-cooled structures 835 of theliquid-cooled electronics rack 800. As noted, and by way of exampleonly, hose barb fittings 850 and quick connect couplings 855 may beemployed to facilitate assembly or disassembly of warm-liquid coolantloop 820.

As with the embodiment of FIG. 4, in one example of the warmcoolant-cooling approach of FIG. 8, ambient temperature might be 30° C.,and coolant temperature 35° C., leaving the air-to-liquid heat exchanger842 of the cooling unit 840. The cooled electronic system depicted thusfacilitates a chiller-less data center. Advantageously, such aliquid-cooling solution provides highly energy efficient cooling of theelectronic systems of the electronics rack using liquid (e.g., water),that is cooled via circulation through the air-to-liquid heat exchangerlocated outdoors (i.e., a dry cooler) with external ambient air beingpumped through the dry cooler. Note that this warm-coolant-coolingapproach of FIG. 8 is presented by way of example only. In alternateapproaches, cooled-coolant-cooling could be substituted for the coolingunit 840 depicted in FIG. 8. Such cold-coolant-cooling might employbuilding-chilled facility coolant to cool the liquid coolant flowingthrough the liquid-cooled electronics rack, and associated air-to-liquidheat exchanger in a manner such as described above in connection withFIGS. 2A-3.

Note further that in the embodiment depicted in FIGS. 8-10, theair-to-liquid heat exchanger comprises a plurality of distinct,coolant-carrying tube sections 817. These coolant-carrying tube sections817 may take any configuration. For example, each coolant-carrying tubesection 817 may be a discrete, sinusoidal-shaped structure having acoolant inlet and a coolant outlet. The coolant inlets may be coupled(in one example) to the coolant supply manifold 830 associated with theair-to-liquid heat exchanger. In an alternate implementation, however,the coolant inlets could be coupled via tubings 819 to thecoolant-cooled structures 835 associated with the electronic systems (ornodes) 836 of electronics rack 800. Note that the concepts disclosedherein are not dependent on the direction of coolant flow through thecooled electronic system. In one implementation, hose barb fittings andquick connect couplings are employed to facilitate assembly ordisassembly of the warm-liquid coolant loop, including the connectionsfor tubings 819 and flexible hoses 900 (FIGS. 9A-10), described below.For example, one end of tubing 819 may comprise a hose barb fitting 850and the other end a quick disconnect coupling 855 for coupling either tothe coolant outlets of the coolant-carrying tube sections 817 of theair-to-liquid heat exchanger or to the coolant inlet of a respectivecoolant-cooled structure 835 associated with an electronic system (ornode) 836 of electronics rack 800.

FIGS. 9A-10 depict in greater detail one embodiment of a cooledelectronic system such as described above in connection with FIG. 8.

As illustrated in FIG. 9A, a plurality of coupling connections 901 maybe provided along rack-level coolant return manifold 831 to facilitatecoupling of coolant exhaust hoses 900 to the return manifold. Thecoolant supply manifold 830 is depicted integrated within theair-to-liquid heat exchanger in FIGS. 9A & 10, and the coolant returnmanifold 831 is disposed within electronics rack 800. By providing anair-to-liquid heat exchanger 815 with multiple discrete coolant-carryingtube sections 817, each with a coolant inlet and a coolant outlet, theair-to-liquid heat exchanger 815 and coolant-cooled structuresassociated with the electronic systems 836 may be coupled in seriesfluid communication as depicted, by way of example, in FIG. 8. Thedistinct, coolant-carrying tube sections 817 may (in one embodiment)each terminate at their coolant outlet in either a quick disconnectcoupler or a hose barb fitting to facilitate coupling of the flexibletubing 819 to the individual coolant-carrying tube sections of the heatexchanger. In one implementation, the flexible tubes 819 are eachconnected to one liquid-cooled node to supply coolant to cool thecoolant-cooled structure associated that one node. After cooling theliquid-cooled structures within the node, the coolant exits from thenode via the coolant exhaust hose 900 to the rack-level coolant returnmanifold 831.

Advantageously, employing a cooling apparatus such as depicted in FIGS.8-10 reduces the number of coolant manifolds required within the cooledelectronic system. Further, the number of coolant-carrying tube sectionscan be readily adjusted to correspond, for example, to the number ofelectronic systems (or nodes) 836 within the electronics rack, or moreparticularly, to correspond with the number of liquid-cooled structures835 to be associated with the electronic systems 836 for facilitatingliquid-cooling of one or more electronic components within those nodes.For example, in an electronics rack 800 comprising 42 electronic systems(or nodes), the heat exchanger may be configured to include 42 distinct,coolant-carrying tube sections, each of which has its own coolant inletand coolant outlet.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprise” (andany form of comprise, such as “comprises” and “comprising”), “have” (andany form of have, such as “has” and “having”), “include” (and any formof include, such as “includes” and “including”), and “contain” (and anyform contain, such as “contains” and “containing”) are open-endedlinking verbs. As a result, a method or device that “comprises”, “has”,“includes” or “contains” one or more steps or elements possesses thoseone or more steps or elements, but is not limited to possessing onlythose one or more steps or elements. Likewise, a step of a method or anelement of a device that “comprises”, “has”, “includes” or “contains”one or more features possesses those one or more features, but is notlimited to possessing only those one or more features. Furthermore, adevice or structure that is configured in a certain way is configured inat least that way, but may also be configured in ways that are notlisted.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.

What is claimed is:
 1. A cooling apparatus for an electronics rack, thecooling apparatus comprising: an air-to-liquid heat exchanger associatedwith the electronics rack and disposed to cool at least a portion of airpassing through the electronics rack, wherein air moves through theelectronics rack from an air inlet side to an air outlet side thereof,and the air-to-liquid heat exchanger comprises a plurality of distinct,coolant-carrying tube sections, each coolant-carrying tube section ofthe plurality of distinct, coolant-carrying tube sections comprising acoolant inlet and a coolant outlet, at least one of the coolant inlet orthe coolant outlet of one coolant-carrying tube section being coupled influid communication with a coolant loop to facilitate flow of coolantthrough the one coolant-carrying tube section; at least onecoolant-cooled structure in thermal contact with at least one electroniccomponent of the electronics rack, the at least one coolant-cooledstructure facilitating transfer of heat from the at least one electroniccomponent to the coolant; and a tube connecting in fluid communicationone coolant-cooled structure of the at least one coolant-cooledstructure, and the other of the coolant inlet or the coolant outlet ofthe one coolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections of the air-to-liquid heat exchanger, thetube facilitating flow of coolant directly between the onecoolant-carrying tube section of the air-to-liquid heat exchanger andthe one coolant-cooled structure of the at least one coolant-cooledstructure.
 2. The cooling apparatus of claim 1, wherein the tube isconnected so that all coolant to pass through the one coolant-carryingtube section of the plurality of distinct, coolant-carrying tubesections of the air-to-liquid heat exchanger also passes through the onecoolant-cooled structure of the at least one coolant-cooled structure.3. The cooling apparatus of claim 1, wherein the coolant inlet of theone coolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections is coupled in fluid communication withthe coolant loop across a coolant supply manifold associated with theair-to-liquid heat exchanger and the coolant outlet of the onecoolant-carrying tube section of the plurality of distinct,coolant-carrying tube sections is connected via the tube to a fluidinlet of the one coolant-cooled structure of the at least onecoolant-cooled structure.
 4. The cooling apparatus of claim 3, wherein afluid outlet of the one coolant-cooled structure of the at least onecoolant-cooled structure is coupled in fluid communication with thecoolant loop across a coolant return manifold associated with theelectronics rack.
 5. The cooling apparatus of claim 1, furthercomprising: an airflow director configured for the electronics rack,wherein the airflow director is configured to redirect airflowexhausting from the electronics rack at the air outlet side thereof viaan airflow return pathway back towards the air inlet side of theelectronics rack, and wherein the air-to-liquid heat exchanger isdisposed within the airflow return pathway for cooling redirectedairflow exhausting from the air outlet side of the electronics rackbefore returning to the air inlet side of the electronics rack.
 6. Thecooling apparatus of claim 5, wherein the air-to-liquid heat exchangeris disposed within the airflow return pathway along a side of theelectronics rack extending transverse to the air inlet side and the airoutlet side thereof.
 7. The cooling apparatus of claim 1, wherein theair-to-liquid heat exchanger is coupled to the electronics rack at theair outlet side of the electronics rack.
 8. The cooling apparatus ofclaim 1, wherein the electronics rack comprises a plurality ofelectronic subsystems, and the cooling apparatus comprises a pluralityof coolant-cooled structures, at least some electronic subsystems eachhaving a respective coolant-cooled structure of the plurality ofcoolant-cooled structures associated therewith, and wherein the coolingapparatus further comprises a plurality of tubes, each tube connectingin fluid communication one coolant-carrying tube section of theplurality of distinct, coolant-carrying tube sections of theair-to-liquid heat exchanger and the coolant-cooled structure associatedwith an electronic subsystem of the at least some electronic subsystemsof the electronics rack to facilitate direct flow of coolant between thecoolant-carrying tube section of the air-to-liquid heat exchanger andthe coolant-cooled structure of the respective electronic subsystemabsent any coolant manifold being between the coolant-carrying tubesection of the air-to-liquid heat exchanger and the coolant-cooledstructure of the respective electronic subsystem.
 9. The coolingapparatus of claim 8, wherein the coolant inlet of each coolant-carryingtube section of the plurality of distinct, coolant-carrying tubesections is coupled in fluid communication with the coolant loop acrossa coolant supply manifold associated with the air-to-liquid heatexchanger, and the coolant outlet of each coolant-carrying tube sectionof the plurality of distinct, coolant-carrying tube sections isconnected in fluid communication with the coolant-cooled structureassociated with a respective electronic subsystem of the at least someelectronic subsystems, and wherein fluid outlets of the coolant-cooledstructures associated with the at least some electronic subsystems arecoupled in fluid communication with the coolant loop across a coolantreturn manifold associated with the electronics rack.
 10. The coolingapparatus of claim 9, wherein the plurality of distinct,coolant-carrying tube sections of the air-to-liquid heat exchangercomprise N distinct, coolant-carrying tube sections and the at leastsome electronic subsystems of the electronics rack comprise N electronicsubsystems.
 11. A cooled electronic system comprising: an electronicsrack comprising an air inlet side and an air outlet side forrespectively enabling ingress and egress of air through the electronicsrack, the electronics rack further comprising at least one electroniccomponent to be cooled; and a cooling, apparatus facilitating cooling ofthe electronics rack, the cooling apparatus comprising: an air-to-liquidheat exchanger associated with the electronics rack and disposed to coolat least a portion of air passing through the electronics rack, theair-to-liquid heat exchanger comprising a plurality of distinct,coolant-carrying tube sections, each coolant-carrying tube section ofthe plurality of distinct, coolant-carrying tube sections comprising acoolant inlet and a coolant outlet, one of the coolant inlet or thecoolant outlet being coupled in fluid communication with a coolant loopto facilitate flow of coolant through the one coolant-carrying tubesection; at least one coolant-cooled structure in thermal contact withan electronic component of the at least one electronic component of theelectronics rack, the at least one coolant-cooled structure facilitatingtransfer of heat from the electronic component to the coolant; and atube connecting in fluid communication one coolant-cooled structure ofthe at least one coolant-cooled structure and the other of the coolantinlet or the coolant outlet of the one coolant-carrying tube section ofthe plurality of distinct, coolant-carrying tube sections of theair-to-liquid heat exchanger, the tube facilitating flow of coolantdirectly between the one coolant-carrying tube section of theair-to-liquid heat exchanger and the one coolant-cooled structure of theat least one coolant-cooled structure.
 12. The cooled electronic systemof claim 11, wherein the tube is connected so that all coolant to passthrough the one coolant-carrying tube section of the plurality ofdistinct, coolant-carrying tube sections of the air-to-liquid heatexchanger also passes through the one coolant-cooled structure of the atleast one coolant-cooled structure.
 13. The cooled electronic system ofclaim 11, wherein the coolant inlet of the one coolant-carrying tubesection of the plurality of distinct, coolant-carrying tube sections iscoupled in fluid communication with the coolant loop across a coolantsupply manifold associated with the air-to-liquid heat exchanger and thecoolant outlet of the one coolant-carrying tube section of the pluralityof distinct, coolant-carrying tube sections is connected via the tube toa fluid inlet of the one coolant-cooled structure of the at least onecoolant-cooled structure, and wherein a fluid outlet of the onecoolant-cooled structure of the at least one coolant-cooled structure iscoupled in fluid communication with the coolant loop across a coolantreturn manifold associated with the electronics rack.
 14. The cooledelectronic system of claim 11, further comprising: an airflow directorconfigured for the electronics rack, wherein the airflow director isconfigured to redirect airflow exhausting from the electronics rack atthe air outlet side thereof via an airflow return pathway back towardsthe air inlet side of the electronics rack, and wherein theair-to-liquid heat exchanger is disposed within the airflow returnpathway for cooling redirected airflow exhausting from the air outletside of the electronics rack before returning to the air inlet side ofthe electronics rack.
 15. The cooled electronic system of claim 14,wherein the air-to-liquid heat exchanger is disposed within the airflowreturn pathway along a side of the electronics rack extending transverseto the air inlet side and the air outlet side thereof.
 16. The cooledelectronic system of claim 11, wherein the air-to-liquid heat exchangeris coupled to the electronics rack at the air outlet side of theelectronics rack.
 17. The cooled electronic system of claim 11, whereinthe electronics rack comprises a plurality of electronic subsystems, andthe cooling apparatus comprises a plurality of coolant-cooledstructures, at least some electronic subsystems each having a respectivecoolant-cooled structure of the plurality of coolant-cooled structuresassociated therewith, and wherein the cooling apparatus furthercomprises a plurality of tubes, each tube connecting in fluidcommunication one coolant-carrying tube section of the plurality ofdistinct, coolant-carrying tube sections of the air-to-liquid heatexchanger and the coolant-cooled structure associated with an electronicsubsystem of the at least some electronic subsystems of the electronicsrack to facilitate direct flow of coolant between the coolant-carryingtube section of the air-to-liquid heat exchanger and the coolant-cooledstructure of the respective electronic subsystem absent any coolantmanifold being between the coolant-carrying tube section of theair-to-liquid heat exchanger and the coolant-cooled structure of therespective electronic subsystem.
 18. The cooled electronic system ofclaim 17, wherein the coolant inlet of each coolant-carrying tubesection of the plurality of distinct, coolant-carrying tube sections iscoupled in fluid communication with the coolant loop across a coolantsupply manifold associated with the air-to-liquid heat exchanger, andthe coolant outlet of each coolant-carrying tube section of theplurality of distinct, coolant-carrying tube sections is connected influid communication with the coolant-cooled structure associated with arespective electronic subsystem of the at least some electronicsubsystems, and wherein fluid outlets of the coolant-cooled structuresassociated with the at least some electronic subsystems are coupled influid communication with the coolant loop across a coolant returnmanifold associated with the electronics rack.